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Figure  Page 

1. )  a).  Normal  VQ  scan  b).  abnormal  VQ  scan.  The  first  row  shows  ventillation  4 

images  and  other  rows  show  perfrusion  images. 

2. )  Two  perfusion  scans:  The  box  shows  overlapped  left  and  right  lungs.  5 

Arrows  point  to  areas  of  low  profusion.  . 

3. )  Cardiac  SPECT  imagery.  Raw  data  -  rows  one  and  two;  PCNN  output-  6 

rows  five  and  six;  cleaned,  enlarged  images  with  defects  marked- 

rows  three  and  four. 

4. )  SPECT  bone  scan  image.  A  physician  has  marked  ROIs  by  drawing  circles.  7 

The  PCNN  locates  the  same  ROI  and  marked  it  with  an  outlined  area  and  filled 

in  an  area  in  red.  c)  In  a  second  patient  scan  the  PCNN  locates  many  cancerous 
lesions  as  ROIs. 

5. )  Optical  input  PCNN  Chip.  9 

6. )  Each  posterior  lung  is  divided  into  two  regions:  lateral  and  medial.  Each  region  1 2 

is  then  subdivided  into  an  upper,  middle  or  lower  region. 

7. )  Concave  defects  at  the  edge  of  the  lung  are  detected  and  their  position  13 

and  size  are  recorded. 

8. )  Lungs  and  the  a)  successful  and  b)  unsuccessful  segmentation  results  14 

9. )  A  Perfusion  lung  that  fragments  into  small  contiguous  areas  rather  than  one  16 

lung.  The  gaps  denote  significant  defects. 

10. )  Examples  of  perfusion  lungs  that  differ  in  size  from  the  ventilation  lungs.  16 

In  this  case  the  left  perfusion  lung  is  also  smaller  than  the  right  lung 

11. )  Undetected  wedge  shaped  defect  (lingula)  located  near  white  arrows.  21 

12. )  Pulse  coupled  neuron  schematic.  This  neuron  is  one  pixel.  The  lightning  bolt  25 

arrow  is  the  optical  input.  This  input  can  be  overridden  in  an  electronic  PCNN  also 

under  development,  b)  The  L-Edit  layout  for  the  pixel/neuron,  c)  tiled  32x32  * 

PCNN  array. 

13. )  Segmented  image  with  Vi  of  array  illuminated .  26 
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Summary 


Pulse  Coupled  Neural  Networks  (PCNN)  have  been  extended  and  modified  to  suit  medical 
image  segmentation  applications.  Previous  research  demonstrated  the  ability  of  a  PCNN  to 
ignore  noisy  variations  in  intensity  and  small  spatial  discontinuities  in  images  that  prove 
beneficial  to  image  segmentation  and  image  smoothing.  This  final  report  describes  three 
research  and  development  projects  that  relate  to  PCNN  segmentation  -  three  different  digital 
image  processing  applications  and  a  CMOS  integrated  circuit  implementation.  The  software  for 
the  diagnosis  of  Pulmonary  Embolism  from  VQ  lung  scans  uses  PCNN  in  single  burst  mode  for 
segmenting  perfusion  and  ventilation  images.  The  second  project  is  attempting  to  detect  cardiac 
infarct  and  ischemia  by  comparing  3-D  SPECT  (Single  Photon  Emission  Computed 
Tomography)  images  of  the  heart  obtained  during  stress  and  rest  conditions,  respectively.  The 
third  application  of  the  PCNN  in  full  body  bone  scans  for  the  detection  of  cancer.  This  research 
transitioned  military  PCNN  technology  to  demonstrate  and  test  algorithms  that  automate  image 
interpretation.  Test  results  were  obtained  from  a  blind  clinical  trial  and  the  algorithm 
categorized  the  patients  based  on  the  quantitative  analysis  of  the  medical  images  physicians 
review. 

These  applications  were  used  to  develop  requirements  for  the  hardware  implementation  of  the 
PCNN.  Two  PCNN  hardware  versions  are  under  development.  One  operates  with  electronic 
input  and  the  second  chip  operated  with  an  optical  input.  The  electronic  input  PCNN  is  still  in 
the  design  and  modeling  stage.  The  experience  gained  in  developing  prototype  chip  confirms 
that  PCNN  can  be  implemented  as  integrated  circuit  at  reasonable  cost.  Further  testing  and  study 
is  needed  to  fully  understand  and  validate  the  chip’s  performance.  Availability  of  a  full  array 
will  benefit  real  time  processing  of  images.  Market  opportunities  exist  for  a  general  prototype 
suitable  for  experimentation  and  also  specialized  chips  for  specific  applications. 
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Introduction 


Advances  in  artificial  neural  networks  that  emulate  biological  systems  have  been  used  for  digital 
image  processing.  However,  most  artificial  neuron  models  fail  to  capture  functional  complexities 
of  their  biological  counterparts.  This  is  one  reason  for  the  limited  success  of  artificial  neural 
networks  in  the  domain  of  image  processing.  Achieving  the  functional  complexities  needed  to 
address  image  processing  through  network  size  and  topology  seems  unsolvable  without 
improving  the  neuron  model.  Recently  significant  progress  has  been  made  toward  the 
development  of  neuron  models  that  resemble  the  cortical  neuron. 

Eckhom  [1990]  performed  experimental  studies  of  a  cat’s  visual  cortex,  and  developed  a  neuron 
model  that  captures  some  of  the  salient  cortical  neuron  features.  Eckhom’s  model  and  its 
variants  are  referred  to  as  pulse  coupled  neurons  (PCN)  and  neural  networks  which  use  pulse 
coupled  neurons  are  called  pulse  coupled  neural  networks  (PCNN).  Ranganath  et  al.  [1994] 
significantly  altered  the  pulse  coupled  neuron  model  to  optimize  its  function  in  image 
processing  applications.  This  pulse  coupled  neural  network  functions  in  a  manner  that  is 
beneficial  to  image  segmentation  and  image  smoothing. 

Kuntimad  [1995]  studied  the  segmentation  capability  of  the  PCNN  and  compared  its 
performance  with  several  commonly  used  methods  ( relaxation  approach,  optimum  thresholding, 
region  growing  methods ).  In  general,  an  image  is  segmented  by  PCNN  by  adjusting  the  values 
of  two  network  parameters  such  that  neurons  corresponding  to  pixels  of  a  region  pulse  together 
and  groups  of  neurons  corresponding  to  adjacent  regions  pulse  at  different  times.  The  two 
network  parameters  are  the  linking  coefficient  and  the  radius  of  the  linking  receptive  field.  If  an 
image  consists  of  two  regions,  object  and  background,  PCNN  may  be  used  in  a  single  burst 
mode  to  separate  object  pixels  from  the  background  pixels.  When  intensity  ranges  of  two 
regions  overlap,  no  thresholding  method  is  able  to  perfectly  separate  one  region  from  the  other. 
However,  depending  on  the  extent  of  overlap,  the  PCNN  is  capable  of  producing  such  a  perfect 
segmentation  result.  Conditions  for  perfect  segmentation  of  two  region  images  are  derived  by 
Kuntimad  [1995].  When  the  two  intensity  levels  overlap  extensively,  even  though  perfect 
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segmentation  is  not  possible,  the  PCNN  performs  remarkably  well.  When  the  image  consists  of 
more  than  two  types  of  regions,  the  PCNN  may  be  used  in  multiple  burst  mode.  Kuntimad 
[1995]  has  developed  a  knowledge-based  system  in  which  PCNN  is  used  in  multiple  burst  mode 
to  segment  complex  images  iteratively.  Each  iteration  improves  the  result  of  the  previous 
iteration  by  updating  network  parameters.  Three  medical  software  applications  and  a  PCNN 
opto-electronic  integrated  circuit  implementation  of  the  PCNN  are  discussed. 

Detection  of  Pulmonary  Embolism 

Valdivia  [1997]  writes  that  pulmonary  embolism  is  a  medical  condition  associated  with 
approximately  250,000  hospitalizations  and  50,000  deaths  annually  in  the  United  States. 
However,  less  than  10%  of  patients  are  treated  for  the  condition  because  of  the  difficulties 
encountered  during  diagnosis.  Medical  doctors  use  VQ  scans  to  infer  or  detect  pulmonary 
embolism.  A  VQ  scan  image  set  consists  of  8  digitally  acquired  perfusion  images  and  more  than 
40  digitally  acquired  ventilation  images  of  the  lungs.  Perfusion  images  monitor  carbon 
dioxide/oxygen  exchange  at  the  lung  exterior  and  the  ventilation  images  monitor  the 
oxygen/carbon  dioxide  exchange  on  the  inside  of  the  lung.  The  diagnostic  data  is  diminished 
intensity  or  voids  that  are  useful  in  identifying  lung  regions  that  do  not  receive  sufficient  blood 
to  support  the  gas  exchange.  These  perfusion  images  are  taken  in  rotational  intervals  of  45  deg. 
All  ventilation  images  cover  the  same  posterior  view  of  the  lungs  and  are  taken  over  a  period  of 
several  minutes  at  regular  time  intervals  of  30  secs.  These  images  are  plagued  by  poor  contrast 
and  low  signal  to  noise  ratio,  figure  1 . 

At  present  doctors  interpret  VQ  scans  visually,  looking  for  intensity  voids  and  hot  spots.  They 
classify  the  patient  into  pulmonary  embolism  categories:  high,  low  or  intermediate.  The  scans 
assigned  to  the  intermediate  category  mean  the  patient  may  or  may  not  have  PE  and  must 
undergo  further  testing  to  confirm  or  rule  out  the  condition.  Obviously  an  intermediate  diagnosis 
is  not  useful.  The  next  test  is  pulmonary  angiography,  a  relatively  expensive  test  that  has  high 
morbidity  and  mortality  rates  associated  with  it.  The  poor  data  quality  and  subjectivity  of  the 
interpretation  causes  a  lot  of  inconsistency  in  VQ  scan  interpretations. 
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The  physician’s  software  tool  for  the  detection  of  pulmonary  embolism  uses  PCNN  in  single 
burst  mode  to  segment  perfusion  and  ventilation  images.  In  each  image,  PCNN  identifies  the 
group  of  pixels  that  form  the  left  and  right  lungs.  The  parameter  values  (linking  coefficient  and 
initial  threshold)  are  derived  from  input  images  without  user  intervention.  All  neurons  with 
primary  input  greater  than  the  initial  threshold  fire  naturally  and  use  linking  signals  to  capture 
neighboring  neurons  in  a  recursive  manner.  Since  the  threshold  signal  generator  is  eliminated 
the  network  operates  quickly.  Regions  of  low  perfusion  do  not  pulse  with  other  lung  regions  and 
appear  as  holes  or  concave  defects  in  segmented  images. 

hi  a  blind  FDA  clinical  study  performed  at  the  University  of  Utah  Medical  School,  the  PCNN 
based  Lung  Scan  Interpretation  tool  was  used  to  perform  fully  automated  computer  diagnosis  of 
patients  that  were  candidates  for  acute  pulmonary  embolism.  One  hundred  and  three  consecutive 
ventilation  /perfusion  (VQ)  lung  scans  for  acute  pulmonary  embolism  (PE)  were  studied.  Three 
nuclear  medicine  physicians  interpreted  the  VQ  scans  using  the  currently  approved  diagnostic 
rules  outlined  in  the  NIH  PIOPED  study.  The  computer  program  used  the  PCNN  to  identify  the 
lung  and  identify  concave  defects  along  the  lung  periphery.  This  criterion  for  diagnosis  is  akin  to 
the  rules  outlined  in  the  PIOPED  logic  was  used  to  correlate  ventilation  and  perfusion  defects 
and  assign  a  score  of  low,  intermediate,  or  high  probability  for  PE.  The  computer  interpretations 
were  then  compared  to  the  interpretations  of  the  physicians.  The  clinical  follow-up  of  all  patients 
at  six  months  consisted  of  medical  record  review  for  the  results  of  pulmonary  angiography,  chest 
CT,  extremity  doppler  ultrasound,  and  repeat  lung  scans.  Sufficient  clinical  data  for  81  patients 
was  collected  and  analyzed  in  this  evaluation.  The  diagnosis  was  confirmed  by  a  review  of 
patient  records  during  a  six  month  follow-up,  table  1 . 

The  computer  software  correctly  diagnosed  70  of  81  (86%)  VQ  scans.  There  were  two  false 
negative  studies  (normal  or  low  probability  patients  with  PE)  and  four  false  positive  studies 
(high  probability  without  PE).  The  computer  classified  only  five  scans  to  the  intermediate 
category.  In  comparison,  the  physician  interpretation  was  correct  in  nine  of  15  patients  with  PE 
and  39  of  66  patients  without  PE.  The  physicians  interpreted  40%  of  the  studies  as  intermediate 
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while  the  computer  interpreted  only  6%  as  intermediate.  In  order  to  achieve  these  results,  the 
PCNN  algorithm  was  used  to  separate  overlapping  lungs  as  shown  in  figure  2.  The  first  pass  of 
the  PCNN  located  the  boundary  incorrectly.  Iterative  processing  with  varying  beta  produced  the 
correct  segementation  result.  The  lingula  (small  segmental  defect)  was  successfully  located  by 
the  PCNN  based  software. 

Table  1 .  Lung  Scan  Interpretation  using  PCNN  Segmentation 


15  Patients  with  PE  66  Patients  without  PE 


Diagnosis 

Physicians 

Computer 

Physicians 

Computer 

High 

9 

13 

0 

4 

Intermediate 

5 

0 

27 

5 

Low 

1 

2 

36 

39 

Normal 

0 

0 

3 

18 

a)  b) 


Figure  1:  a)  Normal  VQ  scan  b)  abnormal  VQ  scan.  The  first  row  shows  ventillation  images 
and  other  rows  show  perfusion  images. 
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a)  b) 

Figure  2.  Two  perfusion  scans:  a)  The  box  shows  overlapped  left  and  right  lungs,  b)  Arrows 
point  to  areas  of  low  profusion. 

Detection  of  Infarction  and  Ischemia 

Cardiac  disease,  infarct  and  ischemia,  can  be  diagnosed  with  a  cardiac  Single  Photon  Emission 
Computed  Tomography  (SPECT)  study.  This  data  set  is  digital  and  provides  a  three-dimensional 
reconstruction  of  the  myocardium  at  rest  and  during  stress.  Physicians  view  the  study  as  slices  of 
the  heart  made  in  three  directions,  horizontal  short  axis,  vertical  and  horizontal  long  axes.  The 
physicians  select  planes  through  the  reconstructed  heart  and  interactively  examine  myocardial 
activity.  Various  radionuclides  such  as  thallium-201  are  used  to  improve  image  quality.  Due  to 
superior  diagnostic  property  over  planar  images,  SPECT  images  also  allow  for  the  quantitative 
analysis  heart’s  functionality.  Ranganath  and  Banish  are  developing  a  software  tool  to  assist 
physicians  when  they  screen  patients  for  ischemia  and  infarction.  The  software  compares  the 
imagery  obtained  during  exercise  (stress  profile)  with  the  imagery  obtained  during  rest  (rest 
profile).  Regions  with  reduced  perfusion  in  stress  profile  in  comparison  with  the  perfusion  in 
rest  profile  indicate  ischemia.  Regions  with  reduced  perfusion  in  both  profiles  usually  indicate 
infarction.  The  software  has  to  handle  blurring  due  to  motion,  hot  spots,  noise  and  variation  due 
to  anatomical  structure  of  patients.  Early  study  shows  that  PCNN  is  able  to  detect  regions  that 
indicate  ischemia  or  infarction.  The  neural  network  has  consistently  separated  pixels  of  reduced 
perfusion  from  normal  pixels  under  various  imaging  conditions.  It  has  also  succeeded  in 
cleaning  unwanted  clutter  around  myocardium  and  suppressing  hot  spots.  At  present  the 
software  has  correctly  interpreted  39  of  a  total  of  42  data  sets  tried.  Sample  cardiac  imagery  is 


shown  in  figure  3.  The  fust  two  rows  of  data  show  a  partial  set  of  the  raw  SPECT  slices  along 
the  short  axis.  Row  one  is  taken  during  stress  and  row  two  is  taken  during  rest.  Row  five  and  six 
are  obtained  by  enlarging  the  segmented  images.  Rows  three  and  four  display  segmented  images 
in  their  original  intensity  along  with  arrows  pointing  to  defects. 
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Figure  3.  Cardiac  SPECT  imagery.  Raw  data  -  rows  one  and  two;  PCNN  output  -  rows  five  and 
six;  cleaned ,  enlarged  images  with  defects  marked  -  rows  three  and  four. 


Location  of  Bone  Cancer  in  Full  Body  Bone  Scans 

To  illustrate  another  medical  region  of  interest  application.  Figure  4a  shows  a  full  body  SPECT 
bone  scan.  This  scan  is  made  to  diagnose  a  host  of  diseases,  one  of  which  is  bone  cancer.  The 
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data  is  very  noisy  and  therefore  diagnostic  features  can  be  subtle.  Noise  is  caused  by  the 
acquisition  protocol,  patient  motion,  patient  body  mass  and  marker  uptake  variability.  In  Figure 
4b,  the  upper  torso  of  a  man  with  cancer  is  shown.  Cancerous  lesions  in  the  upper  torso  have 
been  identified  in  full  bony  bone  scans.  These  appear  as  bright  spots  on  bones.  However,  not  all 
bright  spots  are  lesions.  The  PCNN  based  region  evaluation  module  sorts  cancerous  lesions 
from  hot  spots  associated  with  arthritis.  The  two  measures  used  to  locate  true  lesions  from  non- 
cancerous  areas  were  mean  and  standard  deviation  of  the  automatically  selected  region. 
Cancerous  lesions  are  local  peaks,  as  in  the  correlation  surface  automated  target  applications. 
Figure  4c  shows  many  regions  of  interest  (red  areas)  overlaid  onto  the  original  torso  intensity. 
These  regions  need  to  be  ranked  because  there  are  true  and  false  lesions  among  the  PCNN 
segmentation  results.  The  segmentation  results  are  referred  to  as  regions  of  interest  (ROI). 


a)  b)  c) 


Figure  4.  a)  SPECT  bone  scan  image.  A  physician  has  marked  ROIs  by  drawing  circles,  b)  The 
PCNN  locates  the  same  ROI  and  marked  it  with  an  outlined  area  and  filled  in  an  area  in  red.  c) 
In  a  second  patient  scan,  the  PCNN  locates  many  cancerous  lesions  as  ROIs. 
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Hardware  Architecture  Pulse  Coupled  Neural  Network  Chip 

One  practical  issue  in  applying  the  PCNN  algorithm  is  the  processing  speed.  The  PCNN  based 
image  segmentation  systems  are  expected  to  deal  with  a  large  number  of  two-dimensional 
images  and  also  three-dimensional  images.  Therefore,  to  process  images  in  real  or  near  real  time, 
a  PCNN  must  be  implemented  as  a  fast  hardware  circuit  with  externally  controllable  parameters. 
SY  Technology  and  its  commercial  subsidiary  MemsOptical  Inc.  have  designed  and  fabricated 
one  such  chip.  Even  though  full  design  details  are  intentionally  left  out,  the  readers  will  get  a  fair 
idea  about  the  capabilities  of  the  PCNN  integrated  circuit. 

An  1 1  x  1 1  PCNN  prototype  chip  has  been  fabricated.  The  size  of  the  chip  is  2  mm  x  2  mm 
square  and  spacing  between  neurons  is  122  pm.  It  is  fabricated  on  the  HP  0.5  pm  process 
offered  by  the  MOSIS  foundry  service.  The  external  or  primary  input  to  each  neuron  is  optical. 

A  photodiode  converts  light  to  a  voltage.  A  lenslet  array  covers  each  neuron,  and  an  imaging 
lens  assists  the  imaging  of  light  onto  the  chip,  so  it  acts  like  a  camera.  Each  neuron  is  linked  to 
all  other  neurons  in  the  network  by  a  grid  of  resistors.  The  linking  coefficient  and  threshold 
signal  are  global  parameters  and  input  externally. 

The  chip  uses  multiplexed  outputs,  each  neuron  is  addressed  by  its  row  and  column  location. 

The  addressing  scheme  is  same  as  that  used  in  CCD  cameras.  The  chip  outputs  a  binary  image 
at  each  threshold  indicating  the  neurons  fired.  We  have  used  a  linearly  decaying  threshold  signal 
in  our  design. 

A  photodiode  (lower  left  figure  5)  is  light  sensitive,  and  feeds  an  amplifier  and  multiplier  circuit. 
The  linking  and  optical  input  are  summed  and  compared  to  a  threshold  voltage.  This  internal 
activity  value  is  above  the  current  threshold  the  neuron  fires.  The  chip  outputs  a  binary  image  at 
each  threshold  level.  Firing  neurons  are  high  and  non-firing  neurons  are  low. 
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The  11x11  array  is  an  experimental  prototype,  from  which  larger  arrays  will  be  based.  Several 
pins  provide  observation  points  for  various  signals  within  the  circuit.  The  prototype  is  still  in  the 
debugging  phase.  Several  prototypes  of  the  1 1  x  1 1  have  been  made  in  order  to  understand  and 
optimize  the  chip’s  performance  in  the  above  mentioned  applications.  Test  data  is  being 
accumulated  to  demonstrate  the  performance  of  the  chip.  This  data  will  then  be  compared  to  the 
results  of  the  circuit  simulation  to  validate  the  circuit  operation.  To  better  facilitate  chip  design  a 
functional  electronic  model  has  been  developed. 

Methods,  Assumptions,  and  Procedures 

Computer-aided  interpretation  of  diagnostic  images  is  a  subject  of  considerable  interest  in 
many  areas  of  medicine  including  nuclear  medicine,  radiology,  echo  cardiology  and  magnetic 
resonance  imaging.  Image  processing  and  pattern  recognition  methods  which  are  successfully 
used  to  solve  many  diverse  and  challenging  problems,  if  adapted  and  applied  properly,  can 
provide  many  useful  clinical  decision  support  tools  for  physicians. 
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For  example,  the  diagnosis  of  acute  PE  is  difficult  due  to  the  low  sensitivity  and  specificity  of 
the  clinical  signs  and  symptoms.  VQ  scans  are  noninvasive  tests  that  give  the  probability  of  PE 
for  a  given  patient.  Unfortunately  a  large  number  of  scans,  39%(1),  fall  into  the  intermediate 
probability  group.  Although  an  experienced  reader  has  a  high  accuracy  rate  just  using 
"gestault"  (2),  the  less  experienced  reader  must  rely  on  various  criteria  for  interpretation.  These 
interpretation  schemes  are  not  standardized.  In  addition  there  is  significant  interobserver 
variability,  particularly  in  the  low  and  intermediate  probability  categories  (3).  This  paper 
describes  a  fully  automated  computer  program  that  aids  the  nuclear  medicine  physician  to 
interpret  VQ  scans  and  categorize  the  probability  of  PE  as  high,  low  or  intermediate.  Some  of 
the  salient  features  of  the  VQ  scan  interpretation  software  are  given  below: 

1 .  The  software  alerts  the  reader  when  images  have  poor  count  density  and  are  not  suitable  for 
reading. 

2.  The  software  enhances  ventilation  and  perfusion  images  for  better  human  perception. 

3.  All  significant  defects  are  ranked,  marked  and  displayed  on  the  monitor  for  physician’s 
review. 

4.  The  program  generates  a  detailed  report  that  includes  clinical  and  quantitative  description 
of  defects  and  the  decision  logic  used  for  the  final  classification  of  PE. 

5.  The  study  shows  that  the  software  significantly  reduces  the  percentage  of  scans  assigned  to 
the  intermediate  category  without  compromising  accuracy. 

In  each  study  (lung,  cardiac  and  bone)  a  patient  population  who  had  undergone  the  appropriate 
Scans  for  acute  disease  were  included  in  the  study.  In  each  study,  the  scans  were  interpreted  by 
physicians,  according  to  the  appropriate  medical  criteria,  by  three  nuclear  medicine  physicians. 
All  potential  defects  were  identified  with  the  help  of  the  artificial  neural  network  known  as 
pulse  coupled  neural  network.  Regions  of  interest  were  located  and  evaluated  through 
quantitative  analysis  of  each  image  sequence.  Based  on  quantitative  attributes  of  each  type  of 
defect,  matching  relevant  images  with  one  another,  defects  were  ranked  as  large,  medium,  small 
and  insignificant.  Based  on  the  number  of  defects  and  their  severity,  software  assigned  a 
probability  of  high,  intermediate  or  low  for  the  specific  disease.  In  the  case  of  the  lung  study, 
a  six  month  follow  up  of  all  patients  consisted  of  medical  record  review  of  pulmonary 
angiography,  chest  CT,  extremity  doppler  US,  and  repeat  lung  scans.  The  final  diagnosis  for 
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pulmonary  was  confirmed  by  patient  records  and  the  performance  of  the  software  tool  was  then 
compared  with  those  of  physicians.  In  the  case  of  the  other  studies,  agreement  among  by  three 
physicians  was  considered  the  gold  standard  (truth). 

The  lung  code  is  the  basis  for  both  other  algorithms,  in  fact,  the  lung  code  has  been  applied  to 
mine  detection,  ultraviolet  satellite  imagery,  and  automated  target  recognition  correlation 
surfaces.  Once  data  is  processed  through  the  lung  code,  the  small  details  of  detection  are  fine 
tuned  to  be  specific  to  the  data  set.  For  example,  the  initial  version  of  the  lung  software  resulted 
in  six  false  negatives  (patients  without  the  disease  classified  as  high  probability)  and  eighteen 
false  positives  (patients  with  the  disease  classified  as  low  probability).  Based  on  the  analysis  of 
these  results  several  improvements  were  made  to  the  detection  algorithm.  Usually  it  is  the  case 
that  it  is  just  the  segmented  region  algorithm  that  requires  fine-tuning.  In  this  lung  case, 
methods  for  improved  defect  detection  and  classification  included  better  separation  of  left  and 
right  lungs  in  oblique  views,  improved  recognition  of  effects  of  heart  in  anterior  views  and 
diaphragms  in  lateral  views.  These  enhancements  decreased  the  number  of  false  negative  and 
false  positive  studies  to  one  and  zero,  respectively.  In  comparison,  physicians  also  had  one 
false  negative  study.  However,  physicians  classified  33%  of  the  studies  as  intermediate  while 
the  software  interpreted  only  5%  as  intermediate. 

The  study  protocol  included  a  population  of  103  patients  who  had  undergone  VQ  scans  for 
acute  PE.  A  population  of  80  patients  who  had  undergone  full  body  scans  for  acute  bone  cancer 
were  included  in  the  bone  study.  A  population  of  40  patients  who  had  undergone  cardiac  scans 
for  infarct  and  ischemia  were  included  in  the  cardiac  study.  Lung  scans  consisted  of  a  posterior 
ventilation  study  performed  using  370-555  MBq  Xe-133,  followed  by  an  eight  view  perfusion 
study  using  111-185  MBq  TC-99m  MAA  containing  approximately  200,000  to  500, 000 
particles.  Each  perfusion  lung  image  is  a  256X256  array  of  pixels  and  each  ventilation  lung 
image  is  a  128x128  array  of  pixels.  Cardiac  data  is  64x64  and  the  bone  scans  are  256x1024. 
The  radionuclide  imaging  protocol  for  cardiac  including  sestamibi  and  thallium.  The  bone  scan 
protocol  was  not  available  at  the  time  of  the  preparation  of  this  report.  In  all  three  studies,  the 
pixel  intensity  is  stored  as  a  16-bit  unsigned  number. 
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Basis  Lung  Scan  Interpretation  Algorithm 


A  step  by  step  description  of  the  VQ  lung  scan  interpretation  algorithm  is  given  in  this  section. 
Step  1:  The  ventilation  image  sequence  is  processed  to  determine  regions  of  strong  and 
moderate  retention. 

a.  Seven  ventilation  images  at  the  end  of  the  washout  sequence  are  added  form  1M. 

b.  The  first  two  images  in  the  ventilation  sequence  (single  breadth  and  equilibrium 
images)  are  analyzed  and  the  one  with  better  intensity  count  and  homogeneity  is 
selected  as  the  reference  Ventilation  image  for  matching  with  perfusion  images  later 
during  the  analysis.  If  both  images  unsatisfactory  due  to  poor  inhalation,  the  user  is 
warned  of  poor  quality  ventilation  images  and  then  proceeds  with  the  analysis.  In 
this  case  the  diagnosis  may  not  be  reliable. 

c.  The  image  selected  in  step  lb  is  processed  using  PCNN  and  regions  corresponding 
to  left  and  right  lungs  are  determined.  The  PCNN  produces  a  binary  image  in  which 
lung  and  background  pixels  are  bright  and  dark,  respectively. 

d.  Each  lung  is  divided  into  six  regions  as  shown  in  figure  6.  Each  region  is  classified 
as  a  region  of  high,  moderate,  mild  or  insignificant  retention  by  comparing  regions’ 
average  intensity  with  the  average  intensity  of  the  same  region  in  IM. 


Posterior  Perfusion  Posterior  Ventilation  Region  Divisions 


Figure  6.  Each  posterior  lung  is  divided  into  two  regions:  lateral  and  medial.  Each  region  is  then 
subdivided  into  an  upper,  middle  or  lower  region. 


Step  2:  The  boundary  of  each  lung  as  defined  by  the  binary  ventilation  image  is  extracted  and 
stored  using  eight  way  chain  code.  For  each  lung,  several  geometric  attributes  are  computed. 
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These  include  size,  perimeter,  height  and  width  of  the  lung  and  erosion  in  the  medial,  basal, 
apical  and  lateral  regions. 

Step  3:  The  binary  ventilation  image  is  also  used  to  find  defects  or  parts  of  lungs  with 
diminished  radionuclide  concentration.  In  ventilation  image,  pixels  that  belong  to  the  lung  are 
usually  brighter  than  non-lung  pixels.  However,  defective  lung  segments,  if  any,  appear  much 
darker  than  normal  lung  segments.  As  a  result  these  darker  lung  segments  usually  blend  with 
the  background  when  processed  by  the  PCNN.  Therefore,  the  segmentation  of  the  ventilation 
image  using  the  PCNN  may  lead  to  any  of  the  following  results: 

a.  A  lung  with  very  large  defects  near  the  base  or  apex  appears  small  in  size  in  the 
binary  image.  This  is  because  a  large  number  lung  pixels  blend  with  the 
background. 

b.  Some  long  and  narrow  defects  may  cause  a  lung  to  appear  fragmented  in  the  binary 
image. 

c.  Interior  defects  appear  as  holes  and  peripheral  defects  appear  as  concavities  along 
the  boundary  of  the  lung.  Position  and  size  of  each  such  defect  are  determined  and 
stored,  figure  7. 

d.  If  there  are  no  ventilation  defects  then  none  of  the  above  are  true. 


Posterior  Perfusion  Ventilation  Segmented  Perfusion 

Figure  7.  Concave  defects  at  the  edge  of  the  lung  are  detected  and  their  position  and  size  are 
recorded. 

Step  4:  Each  of  the  eight  perfusion  images  are  processed  to  determine  all  potential  perfusion 
defects  and  their  attributes. 

a.  The  image  is  smoothed  using  a  neighborhood  averaging  method  to  reduce  the  ill 
effects  of  random  noise.  Any  lowpass  filter  can  be  used  for  this  purpose. 
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b.  The  smoothed  image  is  applied  as  input  to  the  PCNN  and  segmented  to  produce  the 
bilevel  image  in  which  lung  and  non-lung  pixels  are  set  to  255  and  0,  respectively. 

c.  A  region-growing  algorithm  is  used  to  determine  all  the  regions  present  in  the  image. 
Ideally  each  lung  must  appear  as  one  connected  region.  However,  this  may  not  be 
true  due  to  the  reasons  described  in  step  3.  The  boundary  of  each  region  is  stored 
using  eight-way  chain  code. 

d.  All  regions  that  belong  to  left  lung  are  identified.  Similarly,  all  regions  that  belong 
to  right  lung  are  also  identified. 

e.  Size  and  position  of  all  defects  that  appear  as  interior  holes  and  concavities  along  the 
lung  boundary  are  determined.  Each  defect  is  classified  as  large,  medium,  small  and 
insignificant  based  on  size,  location  and  depth.  For  example,  a  lateral  defect  is  more 
significant  than  a  medial  defect  of  equal  size. 

f.  Basal,  apical,  medial  and  lateral  erosions  are  computed  for  left  and  right  lungs  for 
posterior  and  anterior  views. 

g.  The  pulse  coupled  neural  network  may  not  always  succeed  in  separating  left  lung 
from  right  lung  while  processing  oblique  views.  This  is  especially  true  for  images 
that  represent  anterior  oblique  views.  A  procedure  has  been  developed  to  separate 
left  lung  from  right  lung  if  the  PCNN  fails  to  do  so,  figure  8. 


Segmented  lungs  Segmentation  Error 


a)  b) 

Figure  8.  Lungs  and  the  a)  successful  and  b)  unsuccessful  segmentation  results 
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h.  In  lateral  views,  only  one  lung  is  visible.  Also,  the  diaphragm  may  look  like  a 
concave  defect.  Therefore,  defects  in  this  area  are  further  analyzed  based  on  depth 
and  curvature  of  the  concavity.  It  is  taken  as  a  valid  defect  only  if  certain  specified 
constraints  are  satisfied. 

Step  5:  In  this  step  redundant  defects  and  unsupported  medial  defects  are  deleted.  If  two 
defects  appear  very  close  to  each  other  then  the  smaller  defect  is  considered  as  redundant  and 
deleted.  An  unsupported  medial  defect  is  visible  in  only  one  view  and  is  not  considered 
significant  unless  it  is  large. 

Step  6:  It  is  possible  that  a  defect  may  appear  or  visible  in  more  than  one  view.  All  instances 
of  the  each  defect  are  identified  and  tagged. 

Step  7:  Perfusion  defects  in  posterior  view  are  matched  or  correlated  with  ventilation  defects. 
All  perfusion  defects  that  have  matching  ventilation  defects  are  tagged  and  deleted  from  further 
consideration. 

Step  8:  In  this  step  perfusion  defects  are  deleted  or  downgraded  based  on  retention  in 
ventilation  lungs.  If  strong  retention  is  detected  in  the  region  of  the  lung  that  corresponds  to 
the  position  of  the  defect,  the  defect  is  not  included  in  the  decision  making  process.  If  the 
retention  is  moderate  the  defect  is  downgraded  by  one  step  (large  to  medium,  medium  to  small, 
small  to  insignificant). 

Step  9:  Based  on  the  number  of  defects,  their  attributes  and  quantitative  measures  of  erosion 
(blunting),  final  diagnosis  is  made.  The  software  assigns  high  probability  for  PE  if  one  or  more 
of  the  following  conditions  are  true  and  stops.  Otherwise,  the  program  proceeds  to  step  10. 
a.  Only  one  lung  is  detected  by  the  PCNN  in  the  posterior  or  anterior  perfusion  view 
when  both  lungs  are  clearly  visible  in  the  ventilation  image.  This  covers  the  cases  in 
which  diminished  radionuclide  concentration  is  obvious  in  most  of  the  lung’s 
segments. 
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b.  Left  or  right  perfusion  lung  appears  fragmented  in  the  binary  images  produced  by 
the  PCNN  and  ventilation  lungs  to  not  appear  fragmented,  figure  9. 

c.  The  size  of  one  of  the  perfusion  lungs  in  posterior  or  anterior  view  is  much  smaller 
than  the  other  lung  as  compared  to  the  relative  sizes  of  ventilation  lungs,  and 
excessive  erosion  is  observed  in  the  basal  or  apical  region  of  the  smaller  perfusion 
lung,  figure  10. 

d.  One  or  more  large  unmatched  defects  exist  in  a  perfusion  lung  and  no  defects  or 
retention  is  detected  in  the  corresponding  ventilation  lung. 


Posterior  Perfusion  Segmented  Perfusion 

Figure  9.  A  perfusion  lung  that  fragments  into  small  contiguous  areas 
rather  than  one  lung.  The  gaps  denote  significant  defects. 


Posterior  Ventilation  Segmented  Ventilation 


Figure  10.  Examples  of  perfusion  lungs  that  differ  in  size  from  the  ventilation 
lungs  are  shown.  In  this  case,  the  left  perfusion  lung  is  also  smaller  than  the  right 
lung. 
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Step  10:  If  the  probability  of  PE  is  not  found  to  be  high  in  step  9,  the  left  and  right  ventilation 
lungs  are  ranked  as  good  or  bad.  A  ventilation  lung  is  ranked  as  bad  if  any  one  of  the  following 
conditions  is  true. 

a.  If  one  or  more  large  defects  are  present. 

b.  The  number  of  large  and  medium  defects  in  the  lung  is  greater  than  two. 

c.  The  lung  fragments  when  segmented  by  the  pulse  coupled  neural  network. 

Step  11:  If  left  ventilation  lung  is  classified  as  good  in  step  10,  left  lung  is  examined  for  the 
presence  of  a  lateral  defect  and  medial  defect  (small  or  medium),  approximately  in  the  middle 
third  of  the  lung  with  a  band  of  slightly  darker  pixels  connecting  them.  If  such  defects  and 
pixel  band  are  found  the  probability  of  PE  is  assigned  the  value  high. 

Step  12:  If  the  probability  of  PE  is  not  found  to  be  high  in  previous  steps  and  if  one  or  more  of 
the  following  conditions  are  true  the  scan  is  assigned  to  the  intermediate  probability  category. 

a.  One  perfusion  lung  in  posterior  view  is  significantly  smaller  than  the  other  in 
comparison  with  the  relative  sizes  of  ventilation  lungs  and  there  is  no  significant 
apical  or  basal  erosion. 

b.  A  lung  which  is  ranked  as  good  in  the  ventilation  image  has  three  or  more  small  and 
medium  unmatched  perfusion  defects. 

c.  A  lung  that  is  ranked  as  bad  in  the  ventilation  image  has  six  or  more  unmatched 
perfusion  defects. 

d.  The  ratio  of  medial  erosion  to  area  of  a  lung  in  posterior  perfusion  image  is  greater 
than  0.3  and  also  greater  than  twice  the  ratio  of  medial  erosion  to  lung  area  of  the 
lung  in  posterior  ventilation  image. 

e.  Intensity  analysis  reveals  that  a  region  whose  area  is  greater  than  30%  of  the  area  is 
darker  than  the  rest  of  the  lung.  Only  those  pixels  of  the  input  image  which  are 
identified  as  lung  pixels  by  binary  perfusion  images  are  used  for  area  computation 
and  intensity  analysis. 
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Step  13:  If  the  probability  of  PE  is  not  high  or  intermediate  then  it  must  be  low.  If  both 
ventilation  lungs  are  ranked  as  good  and  no  defects  are  found  in  perfusion  images,  the  lungs  are 
called  normal. 

Basis  of  Cardiac  Scan  Algorithm 

A  step  by  step  description  of  the  cardiac  scan  interpretation  algorithm  is  given  in  this  section. 
Step  1:  The  cardiac  scan  image  short  axis  planar  image  sequence  is  processed  to  determine 
regions  of  the  myocardium. 

a.  rest  images  are  segmented 

b.  stress  images  are  segmented 

Step  2:  The  boundary  of  each  heart  as  defined  by  the  binary  image  is  extracted  and  stored  using 
eight  way  chain  code.  For.  each  heart,  several  geometric  attributes  are  computed.  These 
include  any  deviation  from  the  normal  circumference  of  a  heart. 

Step  3:  If  a  concavity  is  located  in  the  external  contour  it  is  a  cardiac  defect. 

Step  4:  If  the  location  of  a  concavity  matches  between  the  stress  and  rest  imagery,  then  the 
diagnosis  is  infarct.  If  there  is  a  stress  defect  and  no  rest  defect  at  a  location  then  it  is  a 
diagnosis  of  ischemia. 

Basis  for  Bone  Scan  Interpretation  Algorithm 

In  the  process  for  determining  which  patients  had  cancer  and  which  did  not,  an  image 
registration  algorithm  was  developed.  In  accordance  with  customary  protocol,  we  are  including 
the  pseudocode  for  that  algorithm  here.  Once  registered,  the  PCNN  performs  region  of  interest 
location  via  the  same  single  pass  segmentation  scheme  as  described  above. 

Pseudocode  for  Image  registration  (ref  -  Phase  I  final  report  SYT95-TR-085) 

/*  genetic  algorithm  mapping  algorithm  */ 

Initialize  candidate  mappings  (random  draws) 
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for(  max  iterations ){ 

evaluate  candidate  mappings 
mutate  candidate  mappings 
micro  genetic  algorithm  loop{ 
while  (not  converged ')/ 

evaluate  and  rank  candidate  mappings 
regenerate  population { 

save  best  mapping 

combine  the  best  and  third  best  mappings 
combine  the  second  and  fourth  best  mappings 

} 

} 

} 


} 

/*  lesion  detection  algorithm  */ 


compute  mean  image 
compute  variance  image 

PCNN  segments  regions  with  potential  regions  above  reference  threshold 
determine  texture  of  points  exceeding  threshold 

Initialize  candidate  mappings  -  five  candidate  solutions  are  initialized  using  random 
draws  from  a  uniform  distribution.  The  range  of  mappings  are  such  that  the  control  points  are 
fixed  to  individual  domains  not  overlapping. 

evaluate  candidate  mappings  -  after  the  mappings  are  established,  they  are  evaluated 
with  the  Mean  Squared  Error  (MSE)  with  the  desired  image  region.  However,  before  this  is 
accomplished,  a  smoothing  kernel  is  convolved  with  the  mapped  image.  This  smoothing 
removes  any  voids  created  in  the  stretching  of  the  image. 

mutate  candidate  mappings  -  once  a  good  estimate  is  obtained,  the  remaining  four 
mapping  estimates  are  created  with  randomized  mutations  of  the  good  estimate.  This 
randomization  is  adaptively  updated  to  fine-tune  the  solution  in  later  iterations. 
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micro  genetic  algorithm  loop  -  this  is  the  micro  genetic  algorithm  control  structure.  In 
this  loop,  numerous  cycles  are  performed  until  the  MSE’s  of  the  individual  estimates  are  all 
equal  to  within  a  numerical  error  constant.  Typically,  this  will  be  performed  in  10  to  30 
iterations. 

evaluate  and  rank  candidate  mappings  -  evaluation  is  performed  on  candidate  mappings 
using  the  MSE  functional  previously  defined.  After  the  evaluation,  the  five  candidates  are 
ranked  form  the  best  to  the  worst,  i.e.  1  thru  5.  This  ranking  is  then  used  to  select  which 
estimates  are  combined  to  form  new  ones  in  the  next  iteration. 

save  best  mapping  -  the  best  estimate  is  retained  for  the  next  iteration.  This  process  is 
termed  “elitism”  and  has  been  demonstrated  as  a  key  feature  in  determination  of  an  optimal 
mapping. 

combine  the  best  and  third  best  mapping  -  the  best  and  third  best  mapping  candidates 
are  uniformly  combined  to  create  two  new  candidate  mappings.  It  is  important  to  note  that 
combining  two  estimates  which  are  ranked  next  to  each  other  causes  premature  convergence  in 
the  population  of  mappings.  To  circumvent  this,  alternate  pairs  are  combined,  i.e.  1  with  3,  and 
2  with  5. 

A  step  by  step  description  of  the  bone  scan  interpretation  algorithm  is  given  in  this  section. 

Step  1:  The  bone  scan  image  is  processed  to  determine  regions  of  high  radioactivity. 

Step  2:  The  location  of  the  regions  of  interest  are  matched  to  typical  signals  from  a  reference 
body.  The  reference  body  was  used  during  the  image  registration  process. 

Step  3:  If  the  location  of  a  concavity  matches  between  the  reference  and  patient  imagery,  then 
the  diagnosis  is  not  cancer.  If  there  is  a  mismatch  between  the  region  of  interest  and  the 
reference  region,  the  diagnosis  is  cancer. 

Results  and  Discussion 

The  first  software  package  developed  was  a  medical  application  was  lung  scan  diagnosis.  A  set 
of  70  YQ  scans  provided  by  the  University  of  Utah  medical  center.  Salt  Lake  City,  were  used  as 


20 


examples  to  develop  this  lung  software.  After  extensive  discussions  with  two  nuclear  medicine 
physicians,  the  first  version  of  the  software  was  developed.  As  mentioned  earlier  a  set  of  103 
VQ  scans  were  then  used  for  testing  the  software.  The  software  called  six  scans  normal  or 
assigned  low  probability  for  PE  while  angiograms  confirmed  PE.  Also,  eighteen  scans  were 
incorrectly  called  high.  In  short,  the  first  version  resulted  in  six  false  negatives  and  eighteen 
false  positives. 

After  reviewing  those  false  negative  and  false  positive  patients,  physicians  identified  the 
following  five  problems  caused  the  false  positives  and  false  negatives. 

1)  In  some  cases,  while  processing  oblique  views  (mostly  in  anterior  oblique  views),  the 
PCNN  was  not  able  to  separate  left  and  right  lungs  completely.  As  a  result,  the  analysis 
algorithm  detected  large  concave  defect  in  the  region  where  the  two  lungs  join  and 
incorrectly  assigned  high  probability  for  PE. 

2)  The  shadow  of  the  heart  visible  in  anterior  view  was  treated  as  a  defect  by  the  software 
when  the  heart  was  large. 

A* 

3)  The  normal  diaphragmatic  contours  in  lateral  views  were  sometimes  taken  as  concave 
defects. 

4)  The  software  was  unable  to  detect  wedge  shaped  defects  that  appear  as  medial  defects  in 
posterior  and  anterior  views,  figure  1 1. 


Figure  11.  Undetected  wedge  shaped  defect  (lingula)  located  near  white  arrows. 
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5)  In  a  few  cases,  large  defects  were  classified  as  small  or  medium  defects  that  lead  to  false 
negatives. 

Based  on  the  above  study  several  improvements  were  made  to  the  algorithm.  These  included 
methods  for  improved  defect  detection  and  classification,  better  separation  of  left  and  right 
lungs  in  oblique  views,  improved  recognition  of  effects  of  heart  in  anterior  views  and 
diaphragms  in  lateral  views.  These  enhancements  decreased  the  number  of  false  negative  and 
false  positive  studies  to  one  and  zero,  respectively.  In  comparison,  physicians  also  had  one 
false  negative  study.  However,  physicians  classified  33%  of  the  studies  as  intermediate  while 
the  software  interpreted  only  5%  as  intermediate.  This  software  package  is  being  marketed  and 
the  FDA  application  process  is  underway. 

VQ  scan  interpretation  has  several  limitations  including  a  large  number  of  intermediate 
probability  scans,  non-standardized  interpretation  schemes,  and  significant  interobserver 
variability.  We  have  a  developed  a  fully  automated  computer  software  which  aids  the  nuclear 
medicine  physician  to  interpret  VQ  scans  and  categorize  the  probability  of  PE.  The  final 
version  of  the  software  performed  as  well  as  the  physician.  In  addition,  the  algorithm  uses  a 
standardized  quantitative  interpretation  scheme,  eliminates  interobsever  variability  and  greatly 
decreases  the  number  of  intermediate  probability  studies.  By  decreasing  the  number  of 
intermediate  studies,  the  program  helps  the  referring  physician  and  improves  patient  care  by 
decreasing  the  need  for  further  evaluation  such  as  pulmonary  angiogram. 

In  an  identical  manner,  cardiac  and  bone  scan  interpretation  interpretation  programs  were 
developed.  The  programs  were  based  on  an  initial  data-set  and  then  the  program  was  refined 
with  additional  data.  Both  of  these  programs’  exhibited  over  85%  correct  diagnosis  on  the 
limited  data  provided.  More  than  150  scans  are  required  to  develop  and  test  the  lung  code.  Less 
than  that  number  were  obtained  for  these  other  applications.  However,  in  the  pursuit  of 
commercial  markets  we  will  continue  to  develop  these  programs  and  a  host  of  diagnostic  tools 
from  this  first  tool. 
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Why  use  a  Pulse  Coupled  Neural  Network 

There  are  two  main  reasons  for  the  success  of  the  VQ  scan  interpretation  software.  First,  the 
interpretation  algorithm  has  successfully  integrated  knowledge  of  three  nuclear  medicine 
physicians.  Secondly,  the  pulse  coupled  neural  network  is  able  to  segment  images  with 
sufficient  accuracy  to  permit  the  detection  and  quantification  of  defects.  It  is  obvious  that  the 
accuracy  of  the  interpretation  system  depends  on  the  accuracy  of  the  image  segmentation 
system.  Poor  intensity  contrast  and  noise  appear  to  have  less  adverse  effect  on  the  performance 
of  the  PCNN  than  on  the  performance  of  human  experts  and  other  image  segmentation 
methods.  Perhaps,  this  is  the  primary  reason  for  the  decrease  in  the  number  of  intermediate 
studies  achieved  by  the  software. 

Eckhom  developed  a  pulse  coupled  neuron  (PCN)  is  an  artificial  neuron  designed  to  resemble 
biological  neurons  found  in  cat’s  visual  cortex.  Networks  of  pulse  coupled  neurons  are  known 
as  pulse  coupled  neural  networks  (PCNN).  Ranganath  then  developed  two-dimensional,  single 
layered  and  laterally  connected  pulse  coupled  neural  networks  are  found  suitable  for  image 
analysis,  particularly  for  image  segmentation  and  image  smoothing.  The  PCNN  is  able  to 
bridge  small  spatial  gaps  and  minor  intensity  variations  in  spatial  data  or  images.  This  property 
makes  them  highly  desirable  for  the  segmentation  of  digital  images.  A  complete  mathematical 
description  of  the  PCN  model  and  PCNN  is  beyond  the  scope  of  this  paper.  Therefore,  a  simple 
description  of  the  structure  PCN  and  operation  of  the  PCNN  is  presented  in  the  next  section. 

Hardware  Implementation  of  the  Pulse  Coupled  Neural  Network 

A  pulse  coupled  neuron  consists  of  three  major  parts:  a  pulse  generator,  a  threshold  signal 
generator  and  a  linking  receptive  field.  Each  neuron  receives  an  external  input  X(t)  and 
monotonically  decaying  linking  inputs  from  all  neighboring  neurons  that  have  fired.  The  initial 
strength  of  linking  input  from  one  neuron  to  another  is  inversely  proportional  to  the  square  of 
the  distance  between  them.  The  linking  receptive  field  gathers  linking  inputs  from  individual 
neurons  to  produce  the  net  linking  input  L(t).  The  internal  activity  of  the  neuron  denoted  by 
U(t)  is  analogous  to  membrane  potential  of  biological  neuron.  It  is  computed  as  the  product  of 
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X(t)  and  (l+aL(t)),  where  a  is  a  positive  constant  known  as  linking  coefficient.  In  addition  to 
U(t),  the  pulse  generator  receives  a  second  input  known  as  threshold  signal  (T(t)).  The  threshold 
signal  may  remain  constant  or  decay  monotonically  depending  on  the  application  and  mode  of 
network  operation.  In  any  case,  when  the  value  of  the  internal  activity  is  greater  than  the  value 
of  the  threshold  signal  the  neuron  fires.  Now,  the  pulse  generator  sends  a  narrow  pulse  on  Y(t), 
the  neuron’s  output.  As  soon  as  the  neuron  fires,  linking  inputs  are  sent  to  its  neighbors  and 
the  threshold  signal  is  charged  and  clamped  to  a  high  value.  This  disables  the  neuron  from 
further  firing  unless  it  is  reset. 

Now  consider  a  simple  network  of  two  neurons  in  which  only  neuron  N1  is  allowed  to  send 
linking  input  to  neuron  N2  (Figure  2).  Let  XI  and  X2  be  the  external  inputs  to  N1  and  N2, 
respectively.  Assume  that  XI  >  X2.  If  77  and  T2  are  both  initialized  to  a  value  Tmax  greater 
than  XI  and  allowed  to  decay,  N1  will  fire  first  and  send  a  linking  input  to  N2.  Because  of 
the  linking  input  the  value  of  U2  increases  from  X2  to  X2(l  +aL2 ).  If  U2  does  not  exceed  T2 
then  N2  will  fire  at  a  later  time.  If  U2  exceeds  T2  then  N2  will  fire  with  Nl.  Now,  N1  has 
captured  A 12.  In  a  general  network,  the  fire  and  capture  process  begins  with  afew  naturally 
firing  neurons  and  continues  recursively.  This  process  in  which  pulsing  neurons  cause  other 
neurons  to  pulse  by  sending  linking  inputs  in  a  recursive  manner  is  known  as  the  capture 
phenomenon. 

The  PCNN  chip  accepts  input  through  a  differential  pair  of  photo-detectors  that  signal  is 
amplified.  The  input  intensity  is  multiplied  with  a  linking  input  from  a  local  neighborhood.  The 
resultant  signal  is  input  to  a  comparator  for  the  intensity  thresholding.  A  D  flip-flop  stores  the 
pixel  state.  These  components  are  all  integrated  into  a  pulse  coupled  neuron  (PCN).  Each 
neuron  is  a  pixel  on  the  camera  focal  plane.  The  PCN  circuit  is  shown  in  figure  12.  These 
PCNs  are  tiled  to  form  arrays.  The  CMOS  Layout  for  that  PCN  pixel  is  122  pm  in  size  and 
shown  in  Figure  6b.  The  layout  for  a  single  pixel  is  much  larger  than  for  a  traditional  camera 
because  of  the  on-the-focal-plane  processing  that  must  occur.  Optical  sensitivity  issues  are 
easily  resolved  by  altering  the  gain  of  the  photodiode  [Ampgain],  and  by  mating  the  PCNN  to  a 
custom  microlens  array.  This  microlens  array  will  be  designed  by  SY  Technology,  Inc.  and 
fabricated  at  the  commercial  company  MemsOptical.  MemsOptical  is  a  Phase  in  partner  for  our 
innovative  research  and  development  efforts. 
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b)  c) 

Figure  12.  a)  Pulse  coupled  neuron  schematic.  This  neuron  is  one  pixel.  The  lightning  bolt  arrow 
is  the  optical  input.  This  input  can  be  overridden  in  an  electronic  PCNN  also  under 
development,  b)  The  L-Edit  layout  for  the  pixel/neuron,  c)  tiled  32x32  PCNN  array. 

During  this  research  effort  a  number  of  prototype  chips  were  fabricated  to  prove  the  PCNN  on 
chip  processor  concept  and  verify  the  operating  parameters  of  each  functional  component.  Table 
2  is  a  summary  of  the  test  results  on  a  component  by  component  basis. 
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Table  2  PCNN  Prototype  Test  Results 


Component 

Comments 

Phototransistor 

Low  dark  current,  alternatives  with  variable  dynamic  range  under 

test 

Multiplier 

Good  gain  control  over  range  of  Beta  =  .8  to  1.2  volts  -  Need  Higher 
Gain  for  some  commercial  applications 

Amplifier 

Good  gain  control  over  range  of  Amplifier  gain  =  .8  to  1.2  volts 

Current  summer 

Output  range  from  2.0  to  4.5  volts 

Comparator 

Switches  normally  with  high  speed 

Multiplexer 

Prevents  multiple  triggering  of  neuron  as  required 

D-Flip-Flop 

Stores  neuron  trigger  state  as  required 

Exclusive  NOR 

Functions  normally 

Output 

transistor 

Drives  resistive  grid  normally 

Resistive  grid 

Observed  Linking  -  Increase  Linking  Voltage 

The  prototypes  were  illuminated  with  various  test 

images  and  the  segmentation  of  the  optical  input 

was  observed.  In  this  case  a  light  and  dark 

pattern  is  imaged  onto  the  array.  The  chip 

exhibited  a  staggered  segmentation  result,  figure 

13.  This  is  due  to  interconnectivity  noise  that 

was  not  detected  in  the  component  by  component 

test  procedures.  The  source  of  the  noise  has  been 

located.  The  test  board  and  multiplexed  signal 

currents  were  a  source.  In  addition  a  new  ground  _  ,  _  _ 

Figure  13.  Segmented  image  with  Vi 

plane  design  has  been  submitted  to  improve  the  of  array  illuminated 

performance.  The  chip  is  due  to  be  submitted  at  the  end  of  May  1999  and  should  be  received  at 

SY  Technology  Inc.  in  10  weeks. 
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Recommendations 


Among  the  outcomes  of  this  research,  the  lung  scan  program  was  developed  as  a  comprehensive 
computerized  method  of  analyzing  Xenon  ventilation  and  perfusion  scans  for  pulmonary 
embolism.  This  analysis  aids  in  the  detection  and  localization  of  unmatched  pulmonary  defects, 
which  are  indicators  of  pulmonary  embolism.  The  lung  scan  program  was  developed  to  operate 
in  a  manner  similar  to  the  Cedars-Sinai  quantitative  thallium  analysis  program. 

The  lung  scan  program  highlights  the  unmatched  pulmonary  defects  as  an  overlay  on  the 
ventilation  and  perfusion  imagery.  First,  the  software  program  automatically  processes  each  of 
the  lung  scan  images.  Then  the  software  program  locates  and  computes  each  lung’s  size.  From 
each  of  the  lung’s  shape  and  size,  the  perfusion  and  ventilation  defects  are  located.  The  software 
program  locates  retained  radioactivity  in  the  three  minute  washout  image.  From  this  data  the 
software  program  computes  a  probability  of  pulmonary  embolism. 

The  user  may  accept  this  diagnosis  if  there  are  no  mitigating  defects  on  the  Chest  X  Ray  or 
contradicting  evidence  in  the  patient  history.  To  date  Chest  X  Ray  findings  are  not  included  in 
the  diagnostic  criteria.  The  software  relies  on  the  physician  running  the  program  to  take  this  X 
Ray  data,  and  patient  history  into  account  when  interpreting  the  graphical  display. 

Fully  automated  computer  programs  were  demonstrated  for  a  number  of  medical  scan 
interpretation  applications.  The  pulse  coupled  neural  network  used  to  segment  images  in 
defense  and  space  applications  was  found  to  be  suitable  for  the  segmentation  of  digital  medical 
scans.  The  clinical  studies  with  actual  patients,  and  software  running  in  a  medical  hospital  has 
shown  that  the  software  performs  as  well  as  the  nuclear  medicine  physician.  In  addition,  the 
algorithm,  by  using  a  standardized  quantitative  interpretation  scheme,  eliminates  interobserver 
variability,  greatly  decreases  the  number  of  intermediate  probability  studies,  and  thereby 
improves  patient  outcomes. 
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Mapping  the  PCNN  algorithm  to  an  opto-electronic  chip  proved  possible  and  results 
demonstrate  the  basic  operation  of  the  prototype.  The  integrated  circuit  has  been  fabricated  a 
number  of  times  and  is  susceptible  to  noise.  This  noise  causes  less  than  ideal  segmentation 
results.  The  next  generation  chip  has  been  modified  to  improve  operating  characteristics.  Large 
array  sizes  will  be  available  in  the  near  future. 
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